Skip to content

Fix llama-server and llama-bench - Stateless approach #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 171 commits into from
Closed

Fix llama-server and llama-bench - Stateless approach #13

wants to merge 171 commits into from

Conversation

@wine99
Copy link
Collaborator

@wine99 wine99 commented Oct 22, 2025

No description provided.

YangleiZouIntel and others added 30 commits October 15, 2025 13:03
@YangleiZouIntel @wine99
Add ggml-openvino base files
675085b
@zhanmyz @wine99
add openvino as optional backend for Llama.cpp ggml
5b28b4f
@zhanmyz @wine99
* Configure the device(default CPU) that uses OpenVINO to compile th…
8a54dfd 
…e model

 * Add OpenVINO ADD operator to Llama.cpp. The output is somewhat abnormal and needs further debugging.
@zhanmyz @wine99
Solve the issue of abnormal model output caused by using OpenVINO ADD…
0fc7124 
… operator
@zhanmyz @wine99
Add OpenVINO MUL operator to GGML of Llama.cpp.
15b6a4f
@zhanmyz @wine99
Add compile options
f246d24
@zhanmyz @wine99
add OpenVINO frontend convert process steps
f7189b5
@zhanmyz @wine99
add get openvino available ops function
d2a306f
@yumengbo @wine99
Add PoC of integration of openvino frontend. Main changes: ggml-ov-fr…
746bd53 
…ontend-utils, GraphIterator, Decoder
@yumengbo @wine99
Implement GgmlOvDecoder. Add dump functions.
047a771
@yumengbo @wine99
Convert subgraph with add, sub, mul, div op to ov model and do infer …
21aaa5f 
…on openvino device
@yumengbo @wine99
Add GGML_OV_FRONTEND option. Add readme.
d3f5b62
@yumengbo @wine99
Change output for infer request to set output tensor. Support scale, …
eedde64 
…view op.
@zhanmyz @wine99
add GET_ROWS operator of OpenVINO to GGML of llama.cpp
6ee5ee4
@zhanmyz @wine99
Update build.md and add operation mapping(GGML to OpenVINO)
0515ab8
@zhanmyz @wine99
add the rms_norm operator implemented using OpenVINO to the GGML back…
63327ea 
…end of llama.cpp
@yumengbo @wine99
Fix issue for output memory copy of infer request
8037eb3
@yumengbo @wine99
Change to implementation following pytorch frontend
b111e6a
@yumengbo @wine99
Add support for UNARY SILU op . Fix pytorch impl bugs.
4b91d1f
@yumengbo @wine99
Support Softmax op
923166f
@yumengbo @wine99
Support Softmax op
f783c2a
@yumengbo @wine99
Support ROPE op.
aedb88a
@zhanmyz @wine99
Add support for RMS_NORM OP
67d51bd
@zhanmyz @wine99
Add MUL_MAT,CPY,CONT as operators implemented in OpenVINO for GGML ba…
33cf85f 
…ckend
@zhanmyz @wine99
Move CPY from GGML OV Backend to OV Frontend
cba097b
@zhanmyz @wine99
add implementation of MUL_MAT, CPY, CONT of GGML ops using OV ops
9b74f0f
@zhanmyz @wine99
add implementation of CPY when the output tensor is non-contiguous
68d53bb
@zhanmyz @wine99
add tmp source code files
1dc4ec6
@zhanmyz @wine99
Execute singel CONT operator is OK
d941a56
@zhanmyz @wine99
Execute CONT & VIEW operators in OV Frontend is OK
f670c64
wine99 and others added 20 commits October 15, 2025 16:06
@wine99
Add Q5_K to support phi-3-q4_k_m
96254f9
@wine99
Requantize Q6_K (gs16) to gs32 on GPU
f785c3d
@wine99
Fix after rebasing
e6dca1b
@wine99
Always apply Eliminate_ZP to fix GPU compile issue on some platforms
2413170
@cavusmustafa @wine99
kvcachefusion support
e444b88
@cavusmustafa @wine99
env variable GGML_OPENVINO_DISABLE_SDPA_OPTIMIZATION added
c40b9fe
@cavusmustafa @wine99
Fix for Phi3
c21d664
@wine99
Fix llama-cli (need to run with --no-warmup)
3370915
@wine99
Fix add_sliced_mask; Revert mulmat, softmax; Remove input attention_s…
47843b6 
…ize, iSWA model not working
@wine99
fix after rebasing
63f6bba
@wine99
Fix llama-3-8b and phi3-mini q4_0 NPU
b41f494
@ravi9 @wine99
Update to OV-2025.3 and CMakeLists.txt
d4eb8ec
@wine99
Add OV CI cache
f89292d
@ravi9
Apply CISC review and update CI to OV2025.3
89b8212
@ravi9
Update CI to run OV dep install before build
09ead55
@ravi9
Update OV dockerfile to use OV2025.3 and update build docs
7d8ea73
@wine99
Style: use switch in supports_ops
6cac650
@wine99
Style: middle ptr and ref align, omit optional struct keyword
9e02b3f
@wine99
Fix llama-cli; WIP llama-server
ac57c17
@wine99
Fix: llama-server works with --cache-ram 0; llama-bench works with …
affee8d 
…`-fa 1`
@github-actions github-actions bot added the ggml label Oct 22, 2025
@wine99 wine99 force-pushed the dev_backend_openvino branch from 956dbf7 to d5038aa Compare November 4, 2025 08:51
@wine99 wine99 closed this Nov 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

ggml

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@wine99 @YangleiZouIntel @zhanmyz @yumengbo @virajwad @ravi9 @cavusmustafa